R Notebook

library(tidyverse)
library(magrittr)
library(plotly)
library(forecast)
library(ggplot2)

Data Loading —-

Working with ilinet only as the table broke by age group doesn’t have State info

ilinet_tbl  <- read_csv('input/ILINet.csv') %>% 
  arrange(REGION, YEAR, WEEK)

Preprocessing —- # Check number of unique values per feature

rr map(ilinet_tbl, ~length(unique(.x)))

Drop irrelevant features

rr ilinet_tbl <- ilinet_tbl[, map_lgl(ilinet_tbl, ~length(unique(.x)) > 2)]

Investigate why WEEK has 53 distinct values (a year has 52 weeks)

rr table(ilinet_tbl$WEEK)

TODO verify what is the week 53 (maybe leap year?), dropping for now

rr ilinet_tbl %<>% filter(WEEK != 53)

Coerce numerical features to numeric (this will introduce NAs where value == ‘X’)

rr ilinet_tbl[, 4:7] <- map(ilinet_tbl[, 4:7], as.numeric)

Check number of NAs per feature

rr ilinet_tbl %>% group_by(REGION) %>% summarise_all(function(x){sum(is.na(x))}) %>% View

NAs are mostly divided between Florida and Virgin Islands, let’s

investigate further

rr ilinet_tbl %>% filter(REGION %in% c(‘Florida’, ‘Virgin Islands’)) %>% group_by(REGION) %>% summarise_all(function(x){sum(is.na(x))/length(x)}) %>% View

100% of Florida info is NA, 60% of Virgin Islands info is NA

TODO decide what to do with these states later, dropping for now

rr ilinet_tbl %<>% filter(!(REGION %in% c(‘Florida’, ‘Virgin Islands’)))

Combine YEAR and WEEK info

Train/Test split —-

Use last 3 weeks as Test Set (as data is released w/ 2 weeks lag + 1 week protection

against data release delay)

rr train <- ilinet_tbl %>% filter(PRETTYDATE < max(PRETTYDATE) - lubridate::weeks(3))

test <- ilinet_tbl %>% filter(PRETTYDATE >= max(PRETTYDATE) - lubridate::weeks(3))

 saveRDS(train, 'data/train.rds')
 saveRDS(test, 'data/test.rds')
 write_csv(train, 'data/train.csv')
 write_csv(test, 'data/test.csv')

General overview of some of the regions

plot_ly(train, x = ~REGION, y = ~ILITOTAL,  type = "box")

trainSubSet <- filter(train, REGION %in% c('Alabama', 'California','Hawaii','New York','Virginia'))
sp <- ggplot(trainSubSet, aes(x=trainSubSet$PRETTYDATE, y=trainSubSet$ILITOTAL)) + geom_line()
sp + facet_grid(unique(trainSubSet$REGION) ~ .)

trainSubSet <- filter(train, REGION %in% c('New Mexico', 'Louisiana','Montana','Oklahoma','Vermont'))
sp <- ggplot(trainSubSet, aes(x=trainSubSet$PRETTYDATE, y=trainSubSet$ILITOTAL)) + geom_line()
sp + facet_grid(unique(trainSubSet$REGION) ~ .)